Determining a maximum inscribed size of a rectangle

ABSTRACT

Embodiments perform an iterative process for enlarging a rectangle having a fixed aspect ratio within a convex polygon to find the largest rectangular area within the convex polygon. The iterative process includes detecting an intersection of one or more corners of the rectangle with the convex polygon and adjusting a position of the rectangle based on the quantity of intersecting corners. The iterative growth process continues until a maximum inscribed size of the rectangle has been determined. Some embodiments process images from bracketed photography and crop areas outside the determined maximum inscribed size when combining the images into a single image.

CROSS-REFERENCE TO RELATED APPLICATION

This application is a continuation of U.S. application Ser. No.13/523,926, filed Feb. 12, 2013, the entirety of which is herebyincorporated by reference herein.

BACKGROUND

Existing systems are capable of combining several individual imagestaken in a burst into a single combined image. For example, the existingsystems combine images of the same scene taken during high dynamicrange, flash/no-flash, or other types of bracketed photography. Some ofthe existing systems register the individual images by translationand/or rotation in an attempt to compensate for global motion occurringduring the image capture. The registered images are then overlaid orotherwise intersected to produce a combined image. The combined imageproduced by the existing systems, however, is often non-rectangular inshape. Converting the non-rectangular intersection of the images into arectangular shape having the same aspect ratio as the individual inputimages is difficult with the existing systems.

SUMMARY

Embodiments of the disclosure find the largest crop rectangle within aconvex polygon. A computing device iteratively enlarges a rectangleinscribed within the convex polygon while maintaining a fixed aspectratio for the rectangle. The iterative enlarging continues until atleast one corner of the rectangle first intersects at least one edge ofthe convex polygon. The computing device determines how many cornershave intersected or touched an edge of the convex polygon. The computingdevice identifies a maximum inscribed size of the rectangle by resumingthe iterative enlarging of the rectangle while moving the rectanglewithin the convex polygon based on the quantity of corners touching anedge of the convex polygon. The iterative enlarging continues until themaximum inscribed size has been reached (e.g., the rectangle can nolonger be inscribed within the convex polygon).

This Summary is provided to introduce a selection of concepts in asimplified form that are further described below in the DetailedDescription. This Summary is not intended to identify key features oressential features of the claimed subject matter, nor is it intended tobe used as an aid in determining the scope of the claimed subjectmatter.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is an exemplary block diagram illustrating the intersection ofimages to produce a convex polygon in which a rectangle can beinscribed.

FIG. 2 is an exemplary block diagram illustrating a computing device forprocessing images to identify a maximum inscribed size of a rectanglewithin a convex polygon formed from the images.

FIG. 3 is an exemplary flow chart illustrating operation of thecomputing device to determine the maximum inscribed size of a croprectangle and to optionally crop a convex polygon.

FIG. 4 is an exemplary flow chart illustrating operation of thecomputing device to perform scaling and translation of a rectanglewithin a convex polygon.

FIG. 5 is an exemplary diagram illustrating computation of a translationvector for iteratively moving a rectangle while enlarging the rectanglewhen two corners intersect the convex polygon.

FIG. 6 is an exemplary diagram illustrating computation of a translationvector for iteratively moving a rectangle while enlarging the rectanglewhen one corner intersects the convex polygon at a vertex of the convexpolygon.

FIG. 7 is an exemplary diagram illustrating computation of translationvectors for moving a rectangle when two corners intersect the same edgeof the convex polygon.

Corresponding reference characters indicate corresponding partsthroughout the drawings.

DETAILED DESCRIPTION

Referring to the figures, embodiments of the disclosure determine thelargest, axis-aligned rectangle of a given aspect ratio inside a convexpolygon. In some embodiments, a computing device 202 iterativelyenlarges and adjusts a position of a crop rectangle or other rectanglewithin the convex polygon until the rectangle reaches a maximuminscribed size.

Aspects of the disclosure are operable in a plurality of environments.For example, the computing device 202 operates on input images 102obtained from high dynamic range photography, flash/no-flash imagepairs, and other forms of bracketed photography in which several images102 or frames from a burst capture are combined together to produce asingle image. Aspects of the disclosure are further operable, however,with any form of bracketing including, but not limited to, focusbracketing, depth-of-field bracketing, white balance bracketing, or ISObracketing.

While examples are described herein with reference to finding thelargest rectangle that fits within a given convex polygon, aspects ofthe disclosure are also operable to find the largest size of othershapes within the given convex polygon. For example, some embodimentsmay find the largest square capable of being inscribed within the convexpolygon.

Referring next to FIG. 1, an exemplary block diagram illustrates theintersection of images 102 to produce an exemplary convex polygon 104 inwhich a rectangle 106 can be inscribed. While examples are describedherein with reference to images 102 taken in a burst, to consecutiveimage captures of the same scene, or to other groups of related images102, aspects of the disclosure are operable with any plurality of images102. For example, the images 102 may be taken at approximately the sametime or at different times (e.g., one image 102 of a scene taken duringthe day and another image 102 of the same scene taken at night). In FIG.1, at least two images 102 have been overlaid or otherwise intersectedto produce the convex polygon 104. Areas outside the convex polygon 104have been removed in FIG. 1 for clarity. Embodiments of the disclosureoperate to identify, determine, calculate, compute, or otherwise findthe largest rectangle 106 within the convex polygon 104.

Referring next to FIG. 2, an exemplary block diagram illustrates thecomputing device 202 for processing images 102 to identify a maximuminscribed size of a rectangle within a convex polygon formed from theimages 102. In the example of FIG. 2, the computing device 202associated with a user 204 represents a system for finding a largestcrop rectangle within a convex polygon. The computing device 202represents any device executing instructions (e.g., as applicationprograms, operating system functionality, or both) to implement theoperations and functionality associated with the computing device 202.The computing device 202 may include a mobile computing device or anyother portable device. In some embodiments, the mobile computing deviceincludes a mobile telephone, laptop, tablet, computing pad, netbook,gaming device, and/or portable media player. The computing device 202may also include less portable devices such as desktop personalcomputers, kiosks, and tabletop devices. Additionally, the computingdevice 202 may represent a group of processing units or other computingdevices.

In some embodiments, the computing device 202 has at least one camera206, at least one processor 208, and at least one memory area 210. Thecamera 206 includes any device or apparatus for capturing still images102 or video, with or without audio. The camera 206, or computing device202, may have auxiliary lighting such as a flash for activation whencapturing the images 102. The camera 206, or computing device 202, maybe capable of various configurations or settings for capturing theimages 102. For example, the camera 206 may be capable of capturingimages 102 at multiple exposures, multiple apertures (e.g., F-stops),etc.

While the computing device 202 in FIG. 2 is shown to include the camera206 and to perform functionality such as described in FIG. 3 and FIG. 4,other embodiments are contemplated. For example, aspects of thedisclosure are operable with a digital camera that has a processor, orotherwise communicates with a device that has a processor, forimplementing the functionality illustrated in FIG. 3 and FIG. 4. Inanother example, aspects of the disclosure may be performed by a cloudcomputing system receiving the images 102 from the computing device 202or any other capture device.

The processor 208 includes any quantity of processing units or types ofprocessing entities, and is programmed to execute computer-executableinstructions for implementing aspects of the disclosure. Theinstructions may be performed by the processor 208 or by multipleprocessors executing within the computing device 202, or performed by aprocessor external to the computing device 202. In some embodiments, theprocessor 208 is programmed to execute instructions such as thoseillustrated in the figures (e.g., FIG. 3 and FIG. 4).

In some embodiments, the processor 208 or other components of thecomputing device 202 represents an implementation of analog techniquesto perform the operations described herein. For example, the operationsmay be performed by an analog computing device and/or a digitalcomputing device.

The computing device 202 further has one or more computer readable mediasuch as the memory area 210. The memory area 210 includes any quantityof media associated with or accessible by the computing device 202. Thememory area 210 may be internal to the computing device 202 (as shown inFIG. 2), external to the computing device 202 (not shown), or both (notshown). In some embodiments, the memory area 210 includes read-onlymemory and/or memory wired into an analog computing device.

The memory area 210 stores, among other data, one or more applications212. The applications 212, when executed by the processor 208, operateto perform functionality on the computing device 202. Exemplaryapplications 212 include mail application programs, web browsers,calendar application programs, address book application programs,messaging programs, media applications, location-based services, searchprograms, and the like. The applications 212 may communicate withcounterpart applications or services such as web services accessible viaa network. For example, the applications 212 may represent downloadedclient-side applications that correspond to server-side servicesexecuting in a cloud.

The memory area 210 further stores a plurality of the images 102, suchas image #1 through image #N, and a corresponding plurality of imagecharacteristics 213, such as image #1 characteristics through image #Ncharacteristics. The images 102 may be captured by the computing device202 via the camera 206, or received by the computing device 202 fromanother entity (e.g., a capture device). In embodiments in which theimages 102 are received by the computing device 202, the computingdevice 202 may not have the camera 206. Each of the imagecharacteristics 213 corresponds to at least one of the images 102. Theimage characteristics 213 represent metadata describing the images 102.Exemplary image characteristics 213 include, but are not limited to,values representing size, format, aspect ratio, type, compression type,and compression ratio. The image characteristics 213 may be calculated,discovered, determined, or received by the computing device 202.

Exemplary images 102 include a flash image (e.g., one of the images 102taken while activating a flash or other light source), and a no-flashimage (e.g., one of the images 102 taken without the flash). In someembodiments, a plurality of the images 102 of substantially the samescene may be grouped together or otherwise associated. Each of theimages 102 in such a group may have been taken with different camera 206configurations such as different exposures.

Further, while aspects of the disclosure are described with reference toimages 102 captured by the computing device 202, aspects of thedisclosure are operable with any images 102 from any source orcombination of sources. For example, in some embodiments, the images 102represent frames sampled from a video stream or other video source.

The memory area 210 further stores one or more computer-executablecomponents. Exemplary components include a grow component 214, an edgecomponent 216, and a vector component 218. Operation of these componentsis described below with reference to FIG. 3.

Referring next to FIG. 3, an exemplary flow chart illustrates operationof the computing device 202 to determine the maximum inscribed size of acrop rectangle and optionally crop a convex polygon. At 302, thecomputing device 202 defines a convex polygon encompassing anintersection of the input images 102. The images 102 may be alignedprior to intersection. For example, translation and/or rotation may beperformed on the images 102 (e.g., from their original, as-capturedform). The translation and/or rotation may be part of image registrationto align the images 102 relative to one or more regions of interest(e.g., faces, buildings, user-define regions, etc.) within the images102. In some embodiments, the computing device 202 overlays each of theimages 102 to produce the convex polygon. Areas outside the convexpolygon may be cropped or otherwise ignored. Operation 302 is optionalbecause in other embodiments, rather than the computing device 202defining the convex polygon, another entity provides the convex polygonas input to the computing device 202.

At 304, the computing device 202 iteratively enlarges or grows arectangle within the convex polygon. While enlarging the rectangle, thecomputing device 202 maintains a fixed aspect ratio for the rectangle insome embodiments. The fixed aspect ratio corresponds to a desired aspectratio for a final image. For example, the fixed aspect ratio may bedefined to equal an aspect ratio associated with one or more of theimages 102 used to define the convex polygon.

The computing device 202 maintains the rectangle about an origin or seedpoint within the convex polygon during the iterative growth process. Insome embodiments, the computing device 202 calculates a center ofgravity of the convex polygon and defines the seed point to be thecalculated center of gravity.

The computing device 202 continues enlarging the rectangle until atleast one corner of the rectangle intersects or touches at least oneedge of the convex polygon. The computing device 202 determines how manycorners of the rectangle are touching edges at 306. For example, two,three, or four corners of the rectangle may simultaneously touch edgesof the convex polygon. In another example, just one corner may intersectan edge.

At 308, the computing device 202 resumes the iterative growth orenlarging of the rectangle started at 304 while moving the rectanglewithin the convex polygon based on the determined quantity of cornerstouching the convex polygon. For example, the rectangle may be movedalong a translation vector defined as described below with reference toFIG. 4. The computing device 202 calculates the translation vector basedon the determined quantity of corners touching the convex polygon.

For example, on determining that one corner of the rectangle intersectsthe convex polygon, resuming the iterative enlarging of the rectangleincludes iteratively enlarging the rectangle while fixing a currentposition of the rectangle until another corner intersects the convexpolygon or the maximum inscribed size has been reached. As anotherexample, on determining that two corners of the rectangle intersect theconvex polygon, iteratively moving and enlarging the rectangle includesdefining a translation vector based on the edges intersected by the twocorners and iteratively moving and enlarging the rectangle along thedefined translation vector. In still another example, on determiningthat three corners of the rectangle intersect the convex polygon, thecomputing device 202 identifies two of the three corners that arediagonally opposed to each other and defines a translation vector basedon the edges intersected by two of the three corners, The computingdevice 202 then iteratively moves and enlarges the rectangle along thedefined translation vector.

The computing device 202 continues to iteratively enlarge the rectangleuntil the rectangle can no longer be inscribed within the convexpolygon, or is otherwise about to exceed the convex polygon. Forexample, the computing device 202 stops enlarging the rectangle when thenext growth iteration will result in a corner of the rectangle exceedingthe convex polygon. The computing device 202 stores, presents, and/oroutputs the maximum inscribed size of the rectangle. The maximuminscribed size represents an optimum size of the rectangle given theinput images 102.

At 310, the computing device 202 may optionally crop the convex polygonusing the rectangle at its maximum inscribed size. For example, areaswithin the convex polygon yet outside the rectangle at its maximuminscribed size may be cropped or otherwise ignored.

One or more of the operations illustrated in FIG. 3 may be performed bythe computer-executable components illustrated in FIG. 2. For example,the grow component 214, when executed by the processor 208, causes theprocessor 208 to iteratively enlarge, by a growth step during or witheach of the iterations, a rectangle inscribed within a convex polygonwhile maintaining a fixed aspect ratio for the rectangle and until atleast one corner of the rectangle first intersects at least one edge ofthe convex polygon. The growth step may correspond to any unit ofmeasure, factor, or size. For example, the growth step may correspond toa quantity of pixels. In some embodiments, the size of the growth stepremains fixed throughout each iteration of the rectangle growth process.In other embodiments, the size of the growth step changes during someiterations of the rectangle growth process, as next described.

The edge component 216, when executed by the processor 208, causes theprocessor 208 to determine a quantity of the corners intersecting edgesof the convex polygon. The vector component 218, when executed by theprocessor 208, causes the processor 208 to identify a maximum inscribedsize of the rectangle by resuming the iterative enlarging, by a reducedgrowth step, of the rectangle while moving the rectangle within theconvex polygon based on the determined quantity of the corners and untilthe rectangle can no longer be inscribed within the polygon. The vectorcomponent 218 calculates the reduced growth step with each iteration.

In some embodiments, the reduced growth step corresponds to one-half ofthe growth step used by the grow component 214. For example, a growthstep N may be reduced by one-half, or any other factor, with eachmove-and-enlarge iteration performed by the vector component 218 (e.g.,a growth step of N pixels for one iteration, one-half N pixels for thenext iteration, one-quarter N pixels for the next iteration, etc.). Thisfactor-based growth strategy ensures rapid convergence to the maximuminscribed size. For example, from an algorithm complexity standpoint,the number of iterations to reach the maximum is proportional to log(N).In other embodiments, the growth step may be reduced by a factor thatremains fixed with each iteration. The factor may also be based on asize of the convex polygon.

The vector component 218 may further calculate, based on the determinedquantity of the corners, at least two candidate translation vectors formoving the rectangle during the iterative growth process. The vectorcomponent 218 employs each of the candidate translation vectors inseparate iterative growth processes and compares the inscribed sizesresulting therefrom (e.g., a first inscribed size, a second inscribedsize, etc.). The largest of the inscribed sizes produced in this manneris identified, selected, and/or output. For example, the vectorcomponent 218 may fork, spawn another thread, or serially execute toevaluate each of the candidate translation vectors.

In other embodiments, the vector component 218 employs just one of thecandidate translation vectors. If the size of the rectangle increasesafter an iteration, the vector component 218 continues using the samecandidate translation vector to grow the rectangle. If, however, thesize of the rectangle does not increase after a defined quantity ofiterations (e.g., one or two), the vector component 218 attempts to growthe rectangle using another candidate translation vector. In suchembodiments, additional candidate translation vectors are used only whena first candidate translation vector does not result in an increase insize of the rectangle.

The computing device 202 may also include a user interface component. Insome embodiments, the user interface component includes a graphics cardfor displaying data to the user 204 (e.g., displaying the convex polygonand/or rectangle) and receiving data from the user 204. The userinterface component may also include computer-executable instructions(e.g., a driver) for operating the graphics card. Further, the userinterface component may include a display (e.g., a touch screen display)and/or computer-executable instructions (e.g., a driver) for operatingthe display. The user interface component may also include one or moreof the following to provide data to the user 204 or receive data fromthe user 204: speakers, a sound card, the camera 206, a microphone, avibration motor, one or more accelerometers, a BLUETOOTH brandcommunication module, global positioning system (GPS) hardware, and aphotoreceptive light sensor. For example, the user 204 may inputcommands or manipulate data by moving the computing device 202 in aparticular way.

Referring next to FIG. 4, an exemplary flow chart illustrates operationof the computing device 202 to perform scaling and translation of arectangle within a convex polygon. At 402, the computing device 202finds or otherwise defines a seed point within the convex polygon fromwhich a crop rectangle is to be grown. In some embodiments, thecomputing device 202 calculates a center of gravity for the convexpolygon and defines the seed point to be the calculated center ofgravity.

The computing device 202 iteratively grows, scales, or otherwiseenlarges the rectangle, centered about the seed point, until at leastone corner of the rectangle touches an edge of the convex polygon. Insome embodiments, the computing device 202 iteratively grows therectangle with a dynamic growth step to accelerate the growing process.For example, the computing device 202 initially grows the rectangle insteps of N. When the rectangle first touches the convex polygon, thenext growth iteration occurs at a reduced growth step such as one-halfof N. The computing device 202 reduces the growth step again (e.g., inhalf) after the rectangle touches another edge of the convex polygon.The growth step continues to be reduced in this manner until the largestrectangle that fits within the convex polygon is found, or the growthstep reaches a pre-defined minimum growth step.

If only one corner of the rectangle touches the convex polygon (notshown in FIG. 4), the computing device 202 continues growing therectangle while maintaining contact between the rectangle and the convexpolygon at the touching corner. The point of contact effectively becomesthe seed point, or center of growth. After one or more growthiterations, at least one additional corner touches or intersects an edgeof the convex polygon. That is, the computing device 202 enlarges therectangle until at least two corners of the rectangle are now touchingthe convex polygon at 404.

If four of the rectangle corners are touching edges of the convexpolygon, the computing device 202 stops the iteratively growth processand the current size of the rectangle, or the size of the rectangleduring the last iteration that did not exceed the convex polygon, isreturned as the maximum rectangle size (not shown) and the processterminates.

Otherwise, if two corners of the rectangle touch edges of the convexpolygon at 406, the computing device 202 attempts to move the rectanglewithin the convex polygon to allow for additional growth or enlargementof the rectangle. In particular, the computing device 202 determines atranslation vector or otherwise computes a move direction at 412. Insome embodiments, the computing device 202 measures or calculates anangle α between the two edges of the convex polygon touched by therectangle, and computes the translation vector based on the measuredangle α. For example, referring next to FIG. 5, an exemplary diagramillustrates computation of the translation vector for iteratively movingthe rectangle while enlarging the rectangle. As shown in FIG. 5, thecomputing device 202 determines the direction or orientation of thetranslation vector based on one-half the measured angle α (e.g., thebisection).

At 414, the computing device 202 performs iterative move and growoperations to move the rectangle along the translation vector whileenlarging the rectangle until no further increase in rectangle area ispossible in the direction of the translation vector. For example, thecomputing device 202 iteratively moves the rectangle along thetranslation vector. In some embodiments, the computing device 202iteratively moves and grows the rectangle with the dynamic growth stepas described above to accelerate the growing process. For example, thecomputing device 202 reduces the growth step (e.g., by one-half) eachtime the rectangle touches an edge of the convex polygon. The growthstep continues to be reduced in this manner until the largest rectanglethat fits within the convex polygon is found, or the growth step reachesa pre-defined minimum growth step.

In each iteration, the computing device 202 moves the rectangle by thegrowth step (e.g., possibly resulting in one or both of the touchingcorners to no longer intersect edges of the convex polygon), and thenenlarges the rectangle by the growth step. The move and grow iterationscontinue to increase the rectangle area until the maximum rectangle sizeon the current translation vector is found. If the current translationvector resulted in at least some additional rectangle area at 416, thecomputing device 202 attempts the move and grow operations again, butalong a new translation vector. For example, when at least two cornersonce again touch edges of the convex polygon (e.g., re-performingoperation 406), the computing device 202 computes a new translationvector (e.g., re-performing operation 412) given a current position ofthe rectangle and proceeds to iteratively move and grow the rectanglealong the new translation vector (e.g., re-performing operation 414).For example, another angle a is measured or calculated between the twoedges of the convex polygon touched by the two corners of the rectangle.The computing device 202 computes the new translation vector based onthe measured angle a and performs the iterative move and grow operationsalong the new translation vector.

If the most recent translation vector did not result in any increase inrectangle area at 416, the computing device 202 concludes that nofurther rectangle enlargement is possible and the process terminates.

If three corners of the rectangle touch edges of the convex polygon at408 instead of two corners, the computing device 202 identifies two ofthe three touching corners that are diagonal from each other at 410. Thecomputing device 202 performs the iterative move and grow operations asdescribed above (e.g., at least operations 412, 414, and 416) using thetwo diagonal corners as input.

For some convex polygons, one or more of the rectangle corners mayintersect a vertex of the convex polygon during the iterative growthprocess. In such scenarios, the computing device 202 computes andemploys a plurality of candidate translation vectors in an attempt tomaximize the rectangle size. For example, referring next to FIG. 6, anexemplary diagram illustrates computation of a plurality of translationvectors when one corner of the rectangle intersects the convex polygonat a vertex of the convex polygon. In the example of FIG. 6, thebottom-left corner of the rectangle touches a vertex V₀ of the convexpolygon. The computing device 202 measures or calculates two angles,such as α₀ and α₁, one for each of the edges having an endpoint atvertex V₀. The computing device 202 computes two candidate translationvectors, one for each of the angles α₀ and α₁. As described above withreference to operation of the vector component 218, the computing device202 may execute two separate, iterative move and grow operations, onefor each of the candidate translation vectors to compare the inscribedrectangle sizes resulting therefrom. The larger of the two inscribedrectangle sizes is selected as the maximum inscribed size, and theprocess terminates. In other embodiments, one of the candidatetranslation vectors is chosen and, if the subsequent iterative move andgrow operations result in an increase in size of the rectangle, theother candidate translation vectors are not used.

In still another embodiment, the computing device 202 selects the “best”candidate translation vector for growing the rectangle. The “best”candidate translation vector corresponds to, for example, the vectorthat will enable the rectangle to grow the largest. In some embodiments,the set of candidate translation vectors may be reduced to one bygeometrical analysis. In the example of FIG. 6, the computing device 202ignores α₁ because the rectangle has room to move only in the directionof “widening” lines (e.g., α₀).

Aspects of the disclosure are operable with computing any quantity ofcandidate translation vectors and selecting the candidate translationvector that produces the largest rectangle area. For example, in somescenarios, each of two corners of the rectangle may intersect atvertices of the convex polygon at the same time. In such scenarios, thecomputing device 202 may compute four candidate translation vectors and(1) select the “best” candidate translation vector to use and thenignore the other candidate translation vectors, (2) select any one ofthe candidate translation vectors and ignore the other candidatetranslation vectors if the rectangle grows in size, or (3) fork or spawnfour separate move and grow processes to compare four rectangle areasresulting from the four candidate translation vectors.

Other scenarios are contemplated by aspects of the disclosure. Forexample, two corners of the rectangle may intersect the same edge of theconvex polygon. Referring next to FIG. 7, an exemplary diagramillustrates computation of translation vectors for moving the rectanglein such an example. In each of the example polygons 702, 704, 706, 708in FIG. 6, the computing device 202 computes a translation vector thatis perpendicular to the edge of the convex polygon 702, 704, 706, 708that intersects with the rectangle. In other embodiments, the computingdevice 202 uses any of the two corners that intersect the edge of theconvex polygon as a new center of growth (e.g., similar to the scenarioin which only one corner touches).

ADDITIONAL EXAMPLES

In some embodiments, the computing device 202 manipulates the camera 206or other sensor for low-level optimization of computational photographytechniques such as high dynamic range photography. For example, thecomputing device 202 may execute a scheduler that orders the capture ofmultiple images 102 in a way that optimizes (e.g., reduces) the quantityof sensor resets. An exemplary ordering includes the camera 206capturing the plurality of images 102 by performing a plurality ofsensor readouts during a single exposure of the camera 206. This reducesthe wait time between image 102 captures and reduces and/or eliminatesghosting or other artifacts (e.g., caused by motion) in the images 102.Further, by overlapping the exposures, some embodiments may omitalignment of the images 102.

Aspects of the disclosure contrast with some of the existing systems inwhich the sensor is reset after sensor values are read out at aparticular setting (e.g., exposure). In such embodiments, to take a setof photographs at different exposure times, the sensor accumulates forat least the sum of the exposure times plus the sensor reset times. Forexample, if the exposure times are 100 ms, 200 ms, and 300 ms and thesensor reset time is 30 ms, the total capture time with some of theexisting systems is shown in Equation (1) below.

100+30+200+30+300+30=690 ms   (1)

In contrast, with the cumulative readout embodiment of the disclosure,the exposure times overlap and only a single reset is incurred insteadof a reset after each readout. In this embodiment, the total capturetime is shown in Equation (2) below, which represents a significantreduction from the total in Equation (1) above.

100+100+100+30=330 ms   (2)

In other embodiments, aspects of the disclosure leverage per-pixelreadout circuitry available via complementary metal-oxide-semiconductor(CMOS) processes to preserve, maximize, or opportunistically leverageavailable electron charge. Typically, photons hit the sensors' pixelswhich then convert the energy into free electrons (e.g., photoelectriceffect). The free electrons are fed to a readout circuit viaanalog/digital converters. With existing systems, the electrons are notconserved in the pixel wells during readout. As such, subsequentreadouts using the existing systems are lower than expected and withmore noise.

A typical CMOS sensor has a Bayer pattern mask covering the pixels andallows the user to generate an RGB image for each pixel, as shown inTable 1 below.

TABLE 1 RGB Image Generated by CMOS Sensor for Each Pixel. GREEN REDBLUE GREEN

With aspects of the disclosure, each pixel is dynamically grouped into anew formation depending on the next highest perfect square correspondingto the number of readouts. This formation may be referred to as a “superpixel.” For example, if three readouts are to be performed, the nexthighest perfect square is four. For a readout schedule of 100 ms (A),200 ms (B), and 300 ms (C), such as described in the example above, adynamically assigned layout pattern and readout patterns are shown belowin Table 2.

TABLE 2 Example Layout Pattern and Readout Patterns. Layout Pattern A BA B A B A B A B . . . C A C A C A C A C A . . . A B A B A B A B A B . .. Readout pattern A A A A A . . . at 100 ms (A) A A A A A . . . A A A AA . . . Readout pattern B B B B B . . . at 200 ms (B) B B B B B . . .Readout pattern C C C C C C . . . at 300 ms (C)

The example readout patterns shown in Table 2 above may also beinterleaved such that, for example, the pixels read at 100 ms may beread also at 200 ms and 300 ms via a per-pixel scheduler. The patternsshown in Table 2 above may be extrapolated to any quantity of readoutsand to the corresponding next highest perfect square size of “superpixels.” In some embodiments, the tradeoff is a modification of theresolution in both the horizontal and vertical direction in inverseproportion to the size of the “super pixel.” In the example above, the“super pixel” size is 4 and reduces the effective resolution four times.

In assigning readouts, zero or more spare pixels per perfect squareblock may be proportionally assigned to the shorter exposures to averageout noise, in non-interleaved readout implementations. In interleavedreadout implementations, the free pixels may be used to further compressthe readout schedule.

At least a portion of the embodiments described above may be implementedvia hardware modifications in a backside-illumination process to supportdiscontinuous readout with minimal effect on image quality.Alternatively or in addition, at least a portion of the embodimentsdescribed above may be implemented in software or firmware such ascomputer-executable instructions that rearrange or otherwise definereadout patterns to, at least, reduce and/or eliminate (1) wait timebetween image 102 captures and (2) ghosting or other artifacts in theimages 102.

For example, given a desired quantity of readout times or intervals,aspects of the disclosure dynamically:

-   -   group each pixel into a formation based on the next highest        perfect square corresponding to the number of readouts;    -   assign a layout pattern based on the formation thus defining        readout patterns for each of the readout times; and    -   perform the readouts at the readout times according to the        assigned layout pattern.

At least a portion of the functionality of the various elements in FIG.2 may be performed by other elements in FIG. 2, or an entity (e.g.,processor, web service, server, application program, computing device,etc.) not shown in FIG. 2.

In some embodiments, the operations illustrated in FIG. 3 and FIG. 4 maybe implemented as software instructions encoded on a computer readablemedium, in hardware programmed or designed to perform the operations, orboth. For example, aspects of the disclosure may be implemented as asystem on a chip or other circuitry including a plurality ofinterconnected, electrically conductive elements.

While no personally identifiable information is tracked by aspects ofthe disclosure, embodiments have been described with reference to datamonitored and/or collected from users 204. In such embodiments, noticeis provided to the users 204 of the collection of the data (e.g., via adialog box or preference setting) and users 204 are given theopportunity to give or deny consent for the monitoring and/orcollection. The consent may take the form of opt-in consent or opt-outconsent.

Exemplary Operating Environment

Exemplary computer readable media include flash memory drives, digitalversatile discs (DVDs), compact discs (CDs), floppy disks, and tapecassettes. By way of example and not limitation, computer readable mediacomprise computer storage media and communication media. Computerstorage media include volatile and nonvolatile, removable andnon-removable media implemented in any method or technology for storageof information such as computer readable instructions, data structures,program modules or other data. Computer storage media are tangible,exclude propagated data signals, and are mutually exclusive tocommunication media. In some embodiments, computer storage media areimplemented in hardware. Exemplary computer storage media include harddisks, flash drives, and other solid-state memory. In contrast,communication media typically embody computer readable instructions,data structures, program modules, or other data in a modulated datasignal such as a carrier wave or other transport mechanism and includeany information delivery media.

Although described in connection with an exemplary computing systemenvironment, embodiments of the disclosure are capable of implementationwith numerous other general purpose or special purpose computing systemenvironments, configurations, or devices.

Examples of well-known computing systems, environments, and/orconfigurations that may be suitable for use with aspects of thedisclosure include, but are not limited to, mobile computing devices,personal computers, server computers, hand-held or laptop devices,multiprocessor systems, gaming consoles, microprocessor-based systems,set top boxes, programmable consumer electronics, mobile telephones,network PCs, minicomputers, mainframe computers, distributed computingenvironments that include any of the above systems or devices, and thelike. Such systems or devices may accept input from the user 204 in anyway, including from input devices such as a keyboard or pointing device,via gesture input, and/or via voice input.

Embodiments of the invention may be described in the general context ofcomputer-executable instructions, such as program modules, executed byone or more computers or other devices. The computer-executableinstructions may be organized into one or more computer-executablecomponents or modules. Generally, program modules include, but are notlimited to, routines, programs, objects, components, and data structuresthat perform particular tasks or implement particular abstract datatypes. Aspects of the invention may be implemented with any number andorganization of such components or modules. For example, aspects of theinvention are not limited to the specific computer-executableinstructions or the specific components or modules illustrated in thefigures and described herein. Other embodiments of the invention mayinclude different computer-executable instructions or components havingmore or less functionality than illustrated and described herein.

Aspects of the invention transform a general-purpose computer into aspecial-purpose computing device when configured to execute theinstructions described herein.

The embodiments illustrated and described herein as well as embodimentsnot specifically described herein but within the scope of aspects of theinvention constitute exemplary means for compensating for global motionby determining the maximum inscribed size of the rectangle within theconvex polygon, and exemplary means for compensating for global motionby determining the maximum inscribed size of the rectangle within theconvex polygon defined from a burst capture of the plurality of images102 by the camera 206 for bracketed photography.

The order of execution or performance of the operations in embodimentsof the invention illustrated and described herein is not essential,unless otherwise specified. That is, the operations may be performed inany order, unless otherwise specified, and embodiments of the inventionmay include additional or fewer operations than those disclosed herein.For example, it is contemplated that executing or performing aparticular operation before, contemporaneously with, or after anotheroperation is within the scope of aspects of the invention.

When introducing elements of aspects of the invention or the embodimentsthereof, the articles “a,” “an,” “the,” and “said” are intended to meanthat there are one or more of the elements. The terms “comprising,”“including,” and “having” are intended to be inclusive and mean thatthere may be additional elements other than the listed elements.

Having described aspects of the invention in detail, it will be apparentthat modifications and variations are possible without departing fromthe scope of aspects of the invention as defined in the appended claims.As various changes could be made in the above constructions, products,and methods without departing from the scope of aspects of theinvention, it is intended that all matter contained in the abovedescription and shown in the accompanying drawings shall be interpretedas illustrative and not in a limiting sense.

What is claimed is:
 1. A system comprising: a camera; a memory areaassociated with a mobile computing device; and a processor programmedto: receive a request to capture a plurality of images with the camera;during a single exposure of the camera, executing a plurality of sensorreadouts corresponding to the plurality of images; and applying a singlesensor reset upon execution of the plurality of sensor readouts.